Up to Blog Posts in Data Extraction

Data Extraction

4 Posts tagged with the best_practices tag
Adron Hall

Recently I have been working a lot with the Webtrends DX web services here internally.  Our consulting & services group has been working with various technologies to connect, retrieve data, and provide that data in informational reports to clients and partners.  In the course of working on these various projects I have added some functionality to the Webtrends DX web services SDK and also made some fairly solid in roads on the practices used to develop applications for Excel, Outlook, and other Microsoft Office Applications.  Here is a break down of the progress I have made over the last few weeks.  I hope it is helpful in everyone's efforts to extend, report, and provide analytics data & knowledge to their respective organizations!

 

Webtrends DX SDK - The SDK provides dramatically simplified standard programmatic access to reports, profile listings, report listings, and a mapper object to provide the reports in various object formats.  Recently I have added the ability for the mapper object to take a standard dimensional report object and provide the report as a standard ADO.NET DataTable.  This dramatically simplifies a lot of the functions of sorting, finding, or specifying various parts of the report data or manipulating it for presentation.  In addition this is a more familiar object format for ETL using SSIS or even other ETL tools  out on the market.  The following is an example of the code used to retrieve a report with the SDK and then do a conversion into a standard DataTable Object.

 

User webtrendsUser =
    new User
        {
            AccountName = Resources.Options.Default.Account,
            UserName = Resources.Options.Default.Password,
            Password = EncryptionHelper.Decrypt(Resources.Options.Default.Password)
        };
 
var report = ReportFactory.CreateDimensionalReport("reportIdGoesHere", "profileIdGoesHere", null, null, null,
                                                   null, null, true, webtrendsUser);
Mapper mapper = new Mapper(report);
DataTable dataTable = mapper.MapReportDefinitionToDataTable();

 

In the code above, I have created a Webtrends User object and assigned the appropriate parameters.  In this case I am using the Resources (Options.Settings) available to Windows Application to store and retrieve those settings.  This way the user will not have to enter them every time they want to run the report.

 

I then have the report object being pulled from the ReportFactory Object via the CreateDimensionalReport factory method.  The parameters that are actually needed are the report ID, the profile ID, and the Webtrends User Object.  The null values are other various parameters that can be set but do not particularly need to be set.

 

Then I create a Mapper Object and pass it the report object via the constructor.  This object requires a report object upon instantiation.  I setup the object this way since there always needs to be a report object before any manipulations or mappings can be made.

 

The last step is to declare the DataTable Object, then pull the data table from the builder method MapReportDefinitionToDataTable().

 

For the SDK assembly and code check out the Document Outlining the offering.  In addition over the next couple of weeks I will be posting more information regarding how to use the SDK, best practices, and code snippets for accessing the Webtrends DX Web Services.  There will be additional blog entries and definitely some documentation coming soon (I'm working on it!).  But in the meantime, if you have any questions, comments, or ideas on how we should extend or modify the SDK please let us know here on the Webtrends Developer Forums!

 

I'll have some more entries real soon about best practices, how to code against Outlook or Excel, and more.  So subscribe and stay up to speed on everything Webtrends!  Thanks for reading.

0 Comments Permalink Tags: dx_api, web_services, api, excel, rest_api, web, webtrends, best_practices, rest, webtrends_web_services, data_extraction, services
Adron Hall

I did a blog entry a while back in regards to doing asynchronous web requests against REST based web services, which is what the Webtrends Web Services are built against.  This entry I'll show how to setup a synchronous web request for data against the services.

 

Benifits:

  • The request is made and in a top down fashion the response is returned with the respective string or stream object data needed.
  • Easier to test versus asynchronous web requests.

 

Disadvantages:

  • The request is made, and will pause or "freeze" while waiting for the response.  In a Windows Forms, WPF, Silverlight (or Flash, etc) client application this will cause the user interface to become unresponsive while waiting for the response.

 

So how do we make a synchronous call?  First create a HttpWebRequest object and instantiate it.

 

            var req = (HttpWebRequest)WebRequest.Create(@"https://ws.webtrends.com/beta/ReportService/profiles/?format=json");

 

Next you'll want to setup credentials.

 

            req.Credentials = new NetworkCredential(@"YourWebTrendsAccount\YouUserName", "YourPassword");

Now make sure when the request is made request the server to use compression.  This is entirely transparent and only needed at this single stage of the request.

 

            req.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");

 

Finally set the method of request and other peripheral properties.

 

            req.Method = "Get";

            req.AllowAutoRedirect = true;

 

Once this is done you're ready to submit the request.

 

                WebResponse res = req.GetResponse();

 

To retrieve a simple string object feed through to the end of a the stream object response that is available via the WebResponse object.  This can also prospectively be done with the actualy HttpWebRequest object as it has a GetResponseStream() method.  I've segmented it here for example.

 

                string jsonResponse = (new StreamReader(res.GetResponseStream()).ReadToEnd());

 

The following is the complete code with exception handling.

 

            // Create a request

            var req = (HttpWebRequest)WebRequest.Create(@"https://ws.webtrends.com/beta/ReportService/profiles/?format=json");


            req.Credentials = new NetworkCredential(@"YourWebTrendsAccount\YouUserName", "YourPassword");

            // Add compression request.  IIS will return a compressed data result.

            req.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");

            req.Method = "Get";

            req.AllowAutoRedirect = true;


            // Read the response

            try

            {

                WebResponse res = req.GetResponse();

                string jsonResponse = (new StreamReader(res.GetResponseStream()).ReadToEnd());


            }

            catch (WebException exw)

            {

                WebResponse response = exw.Response;

                string error = (new StreamReader(response.GetResponseStream())).ReadToEnd();

            }

 

Hope that helps!  Leave a comment or question if anything comes up while coding!

 

In a following entry the team will be providing more examples in consumption of these results programmaticaly, but as mentioned by Developer_Donut's entry we'll be releasing a software SDK soon.  This SDK will drammatically simplify and provide guidance on how to access and to utilize the web services.

0 Comments Permalink Tags: web_services, webtrends_web_services, httprequest, json, rest, sample, services, best_practices
Adron Hall

When making requests for reports via the WebTrends Web Services the best practice is to get the data compressed.  When making an HTTP request it's really easy to make sure the web server will compress the data and speed up the transfer dramatically.  So how does one get the content type set to make a request for compressed results programmatically?  Easy, and here's a quick example I put together in C# with the standard HttpWebRequest objects.


        [Test]
        public void TestBaselineCompressedRequest()
        {
            var request = (HttpWebRequest)WebRequest.Create("https://somedomain/theservicepath/to/rest/?format=xml");
            request.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");
            request.Credentials = new NetworkCredential("someUser", "somePassword");

 

            var response = request.GetResponse();
            var responseStream = new StreamReader(response.GetResponseStream(), Encoding.UTF8);
            string getTheResults = responseStream.ReadToEnd();

 

            Assert.IsNotNull(responseStream);
            Assert.IsNotNull(getTheResults);

        }

 

This should get anyone kick started getting compressed data from WebTrends REST Web Services.

0 Comments 0 References Permalink Tags: compression, best_practices, authentication, rest, sample, web_services, webtrends_web_services, http, request
Adron Hall

Today I'm going to cover material around connecting to WebTrends new REST based Web Services.  To start off, let's cover the context behind the methods of making calls to these services.

 

There are two specific ways in which to connect to REST based Web Services, asynchronously and synchronously.  Depending on the type of application, tool, or other mechanism you are connecting to the services with you will want to use one or the other or might be using one or the other.

 

When making an asynchronous call against a service the application, or to be specific the calling thread, will initiate the call but continue executing without waiting for a response.  When the response is received an event fires and the data returned can then be handled in some way.  For client service applications, server based applications, or anything that needs to return control to the application or UI thread, this is the method to use.  This provides a smoother and more continuous application flow when an end user is accessing services.

 

A synchronous call against a service executes on the actual application thread itself, and holding that thread until a response returns.  This can make a UI appear to hang or freeze.  Often this is only used if there is no way to return data via a post event.  One scenario that would need to do this would be to render a page, or display, in its entirety before returning it to the user.  This is a common scenario on the web.  AJAX solves this to some degree, but often one still needs to render the entire initial page, and thus, must wait for the data to return.

 

Best Practice

 

The best practice is to use asynchronous calls.  Only use synchronous calls for stand alone processing or non-user viewed feeds.

 

A Simple Example

 

Here is a C# asynchronous call being made against our WebTrends Web Services.  Keep in mind, I'm providing these as examples, I do NOT suggest writing tests with Thread.Wait calls in them.

 

The example also shows how to setup your WebTrends Credentials for authenticating to our WebTrends Web Services.

 

    [TestClass]
    public class GetRestServices
    {
        private XDocument doc;

 

        private void svc_DownloadStringCompleted(object sender, DownloadStringCompletedEventArgs e)
        {
            doc = XDocument.Parse(e.Result);
        }

 

        [TestMethod]
        public void TestAsynchronousCallWithSecurity()
        {
            const string baseUri = "https://ws.webtrends.com/beta/reportservice/profile/?format=xml";

 

            var svc = new WebClient();
            svc.Credentials = new NetworkCredential("yourWebTrendsAccount\WebTrendsUserName", "yourSuperSecretPassword");
            svc.DownloadStringCompleted += svc_DownloadStringCompleted;
            svc.DownloadStringAsync(new Uri(baseUri));

 

            Thread.Sleep(3000);
            Assert.IsNotNull(doc);
            var docTest = new XDocument();
            Assert.IsInstanceOfType(doc, docTest.GetType());
        }
    }

 

Hope those are helpful. 

0 Comments 0 References Permalink Tags: rest, user_accounts, web_services, webtrends, webtrends_web_services, authentication, best_practices, sample, web_analytics

Actions

Popular Blog Posts