Data Extraction

12 Posts tagged with the rest tag

I have seen a bit of an interest in the original sample application that I put together and since then there have been a number of changes to to DX API.  With the latest release of the Webtrends Analytics Insight user interface we have now update this API to V2.  Along with that I have had a chance to do additional testing of the Excel VBA application.  These two events have now led to version v6 of the Sample Application.

 

So, what’s in this latest version?  Well, first of all, there are bug fixes.  I won’t detail them here, but there were a few and there are probably a few still in there. Next I have updated all of the REST calls to V2 of the API.  And, finally, I have added new functionality that will allow you to see both the Profile and Report Meta data items for any selected report.  It’s all in the documentation, so please read it for details.

 

One of the interesting updates in the V2 DX API is the inclusion of two new reports: an Account Key Metrics report and a Profile Key Metrics report.  These have been added as new reports in this Sample Application as well.

 

So, as always, enjoy this new version of the application and any feedback you have is welcome.  The new version can be found as an embedded object in the attached documentation.

 

Michael Love

Senior Solutions Engineer

0 Comments Permalink

Recently I have been working a lot with the Webtrends DX web services here internally.  Our consulting & services group has been working with various technologies to connect, retrieve data, and provide that data in informational reports to clients and partners.  In the course of working on these various projects I have added some functionality to the Webtrends DX web services SDK and also made some fairly solid in roads on the practices used to develop applications for Excel, Outlook, and other Microsoft Office Applications.  Here is a break down of the progress I have made over the last few weeks.  I hope it is helpful in everyone's efforts to extend, report, and provide analytics data & knowledge to their respective organizations!

 

Webtrends DX SDK - The SDK provides dramatically simplified standard programmatic access to reports, profile listings, report listings, and a mapper object to provide the reports in various object formats.  Recently I have added the ability for the mapper object to take a standard dimensional report object and provide the report as a standard ADO.NET DataTable.  This dramatically simplifies a lot of the functions of sorting, finding, or specifying various parts of the report data or manipulating it for presentation.  In addition this is a more familiar object format for ETL using SSIS or even other ETL tools  out on the market.  The following is an example of the code used to retrieve a report with the SDK and then do a conversion into a standard DataTable Object.

 

User webtrendsUser =
    new User
        {
            AccountName = Resources.Options.Default.Account,
            UserName = Resources.Options.Default.Password,
            Password = EncryptionHelper.Decrypt(Resources.Options.Default.Password)
        };
 
var report = ReportFactory.CreateDimensionalReport("reportIdGoesHere", "profileIdGoesHere", null, null, null,
                                                   null, null, true, webtrendsUser);
Mapper mapper = new Mapper(report);
DataTable dataTable = mapper.MapReportDefinitionToDataTable();

 

In the code above, I have created a Webtrends User object and assigned the appropriate parameters.  In this case I am using the Resources (Options.Settings) available to Windows Application to store and retrieve those settings.  This way the user will not have to enter them every time they want to run the report.

 

I then have the report object being pulled from the ReportFactory Object via the CreateDimensionalReport factory method.  The parameters that are actually needed are the report ID, the profile ID, and the Webtrends User Object.  The null values are other various parameters that can be set but do not particularly need to be set.

 

Then I create a Mapper Object and pass it the report object via the constructor.  This object requires a report object upon instantiation.  I setup the object this way since there always needs to be a report object before any manipulations or mappings can be made.

 

The last step is to declare the DataTable Object, then pull the data table from the builder method MapReportDefinitionToDataTable().

 

For the SDK assembly and code check out the Document Outlining the offering.  In addition over the next couple of weeks I will be posting more information regarding how to use the SDK, best practices, and code snippets for accessing the Webtrends DX Web Services.  There will be additional blog entries and definitely some documentation coming soon (I'm working on it!).  But in the meantime, if you have any questions, comments, or ideas on how we should extend or modify the SDK please let us know here on the Webtrends Developer Forums!

 

I'll have some more entries real soon about best practices, how to code against Outlook or Excel, and more.  So subscribe and stay up to speed on everything Webtrends!  Thanks for reading.

0 Comments Permalink

This is the final installment of the postings on the sample application.  Up to this point we have dealt with the process of getting information about profiles and report definitions.  We have yet to get any real report data.  So, this post will deal with that process and what we do with the report results once we get them.

 

The first step in the process is, of course that of creating a REST URL that requests report data.  The application takes care of all of this based on the selections made by the user for such things as time periods to cover, search criteria, limiting rows, how to deal with totals, and the selection of specific measures.  There is a fair amount of work involved in making sure everything is valid, but the end result is a REST URL that looks something the one shown below.

 

https://ws.webtrends.com/v1_1/ReportService/profiles/YcL4a5dufF6/reports/95df19b6d9f2/?totals=all&period=2009m01d16&format=html&suppress_error_codes=true

 

The major difference between this URL and those we have seen in the past is that there are multiple parameters specified which control what and how much data we get back.  A second difference here is that we are also using a different data type for our results.  In this case we have chosen the “html” format rather than JSON.  When getting report results for Excel the “html” format makes it extremely easy to populate the result data into Excel.  The result data is returned as a table definition which when pasted into a spreadsheet automatically places our data into rows and columns in a tight report fashion.  In fact the result data from the REST URL above produces the results shown below.

 

<table border=1>
    <tr>
        <th>Time</th>
        <th>Browsers</th>
        <th>Views</th>
        <th>Visits</th>
    </tr>
    <tr>
        <td>1/16/2009</td>
        <td>  </td>
        <td>83752</td>
        <td>8941</td>
    </tr>
    <tr>
        <td>1/16/2009</td>
        <td>Microsoft Internet Explorer</td>
        <td>48488</td>
        <td>5193</td>
    </tr>
    <tr>
        <td>1/16/2009</td>
        <td>Mozilla</td>
        <td>13547</td>
        <td>1445</td>
    </tr>
    <tr>
        <td>1/16/2009</td>
        <td>Netscape</td>
        <td>5942</td>
        <td>636</td>
    </tr>
</table>

If you look closely at this result, you will see that we have a single table with a border (width of 1 pixel).  The table is divided into several table rows.  Each table row has a number of columns.  These hold the dimension and measure data for the report.  Also important to note are the first two rows in the table data.  The first row holds the names of the report columns.  In every report I have seen the first column has always been time.  The second row holds the “total” data.  Even when a report has no totals this row of data will be present and will contain the date value.  The remaining rows contain the report detail as specified in the REST URL request.

 

It is these two top rows that may need some special handling.  Suppose I have a series of queries that return report data for individual periods, say one week’s worth of data but one day at a time.  What I really want to get to here is a single report that shows each result for each day, but I only want one set of header data and I don’t want any totals since I can create those myself.  If I were to paste in the results from two successive days without making any changes to the results, I would get a spreadsheet that looks like the following.

 

Time

Browsers

Views

Visits

1/16/2009

83752

8941

1/16/2009

Microsoft Internet Explorer

48488

5193

1/16/2009

Mozilla

13547

1445

1/16/2009

Netscape

5942

636

Time

Browsers

Views

Visits

1/15/2009

93520

9992

1/15/2009

Microsoft Internet Explorer

54057

5821

1/15/2009

Mozilla

15601

1666

1/15/2009

Netscape

6120

635

 

Note that this table contains two sets of headers and two total columns.  What we really want is a result that has only one header row and no totals.  It should look like the following.

 

Time

Browsers

Views

Visits

1/16/2009

Microsoft Internet Explorer

48488

5193

1/16/2009

Mozilla

13547

1445

1/16/2009

Netscape

5942

636

1/15/2009

Microsoft Internet Explorer

54057

5821

1/15/2009

Mozilla

15601

1666

1/15/2009

Netscape

6120

635

 

This gives me a table that I can work with better for creating pivot tables, charts and appropriate totals.

So, how do we do this?  Well, first of all, the application allows you to define that you want this data, headers or totals, stripped off before inserting into the spreadsheet of choice.  Since we know exactly where these are in the result set, we really only need to identify which row of data to remove: headers are always in row one and totals are in row two.  The code in the application that does this is shown below.

 

' Check to see if we need to strip off the Date/Totals row

        If UCase(m_strIgnoreTotalDate) = "Y" Then

            m_intCheck = InStr(1, g_strResult, "</tr>")  ' Find the first end of table row - that's the header row

            If m_intCheck > 1 Then

                m_intStart = m_intCheck + 1                                                 ' Start of date/total row

                m_intEnd = InStr(m_intStart, g_strResult, "</tr>", vbTextCompare) + 4    ' End of date/total row

                ' rebuild the result without the date/total stuff

                g_strResulta = Left(g_strResult, m_intStart + 3)

                g_strResultb = Right(g_strResult, Len(g_strResult) - m_intEnd)

                g_strResult = g_strResulta & g_strResultb

            End If

        End If

       

        ' Get our data and put it on the Clipboard

        ' Check to see if we need to strip off the header row

        If UCase(m_strIgnoreHeader) = "Y" Then

            m_intCheck = InStr(1, g_strResult, "<table border=1>")

            If m_intCheck = 1 Then

                m_intStart = InStr(1, g_strResult, "<tr>")                       ' Start of the header row

                m_intEnd = InStr(1, g_strResult, "</tr>", vbTextCompare) + 4     ' End of the header row

                ' Rebuild our result without that stuff

                g_strResulta = Left(g_strResult, 16)

                g_strResultb = Right(g_strResult, Len(g_strResult) - m_intEnd)

                g_strResult = g_strResulta & g_strResultb

            End If

        End If

 

The right approach is to first look to see if we want to remove the “Total” row and do that.  The reason for this is that we know this data is in row two.  If we were to delete the header row first, the total row would move to the first row and would complicate the logic.  So, the easy way to do this once we have decided that removing it is the thing to do, we simply look for where that row starts and ends.  The start position is simply the first character after the end of the first row and the end is the last position of the “</tr>” string.  Once we calculate those two values we simply recreate our result to copy the data up to the start and after the end of that row.

 

The same basic approach is taken for the header row.  We know exactly where the start of that row is because we know the number of characters leading up to it.  We then calculate the end in the same way we did for the totals row except that we are only looking for the end of this first row.  Once we have these values calculated, we can again recreate the result set to include only the data prior to and after the first row of data.

 

Once we have the result set that is needed for our spreadsheet the only real task left is to decide where to put it.  By that I mean what worksheet and which cell.  The cell we need is the top left cell.  Then we paste our data in there and we are done.

 

I hope these posts have provided useful information.  Whether you choose to use this sample application or to write your own my goal here has been to provide some insight into how one can approach using the REST URLs with Excel and VBA.  I will continue to work on this application with bug fixes (I am sure there are a few) and new functionality.  Eventually I will add the trend reports and will support future versions of the API.

 

As always, I welcome your comments.

 

Michael Love

Senior Solutions Engineer

0 Comments Permalink

The previous posts on this application have provided some insight on how to get general profile and report data.  We have also parsed the results that were returned assuming that we used the JSON data format.  So now we are just about ready to create a REST URL that will allow us to fetch some actual report data.  But before we do there is one last task to be performed and that is getting a list of measures that are available for our desired report.  We need to do that just in case we want to limit the measures return to specific measures rather than all.  One other element we are interested in knowing is whether or not we can use the “search” functionality to limit the results based on contents in the dimension data.  Whether or not we can do that is determined by whether or not the report is multi-dimensional or hierarchical.

 

The REST URL to make this request to get what is referred to as the report meta data is shown below.  This URL clearly defines both the Profile and the Report desired via their ID values.  The “info” field defines this as a request for the report meta data.

 

     https://ws.webtrends.com/v1_1/ReportService/profiles/YcL4a5dufF6/reports/95df19b6d9f2/info/?format=json

 

This request is made in the same way that previous requests are made and were discussed in my prior posts.  The resulting data in JSON format will look something like the data below.  Remember, however, that there are no spaces or other extraneous characters returned.  These are shown here to make this easy to read.

 

{
    "accountID" : 18079,
    "profileID" : "YcL4a5dufF6",
    "ID" : "95df19b6d9f2",
    "name" : "Browsers",
    "language" : null,
    "type" : "dimensional",
    "properties" : {
        "isHierarchy" : false,
        "intervalsEnabled" : false,
        "internalID" : "topbrowsers_v",
        "IsRealTimeCompatible" : true,
        "ProfileCategory" : null
    },
    "dimension" : {
        "ID" : "Desc",
        "name" : "Browsers",
        "type" : "data",
        "Range" : null,
        "Properties" : null,
        "SubDimension" : null
    },
    "measures" : [
        {
            "name" : "Views",
            "accumulationType" : null,
            "ID" : "Hits-0",
            "columnID" : 0,
            "measureFormatType" : "numeric",
            "AllowTotals" : true
        },
        {
            "name" : "Visits",
            "accumulationType" : null,
            "ID" : "Users-0",
            "columnID" : 0,
            "measureFormatType" : "numeric",
            "AllowTotals" : true
        }
    ]
}

 

The entire result set is enclosed in curly brackets – “{  }”.  There are within this set of data three (3) main segments of data.  The first segment is general report information.  Here we have such information as the Profile ID, the Report ID, the Report Name, and other general properties.  It is here that we find a couple of critical pieces of data. These are both in the properties area: isHierarchy”, and "intervalsEnabled".  The “isHierarchy” value tells us whether or not we can use the search functionality.  A value of “false” indicates this is a single dimension report and that we can do this.  On the other hand, a value of “true” indicates that the report contains more than a single dimension and that we cannot use the search functionality.  The second property item that may be of interest to us is the "intervalsEnabled" item.  The value of this item tells whether or not trends data is available and we can use those time periods to retrieve data. Again, a value of “false” means interval/trend data is not available and “true” means that interval/trend data is available.  This sample application has not implemented or enabled this feature, so for our discussion here the point is moot.

 

The second segment of data in our result is a list of dimensions.  In a single dimension report this list is obviously short and appears as shown above.  In a multi-dimensional report the sub dimensions are listed as shown below.

 

"dimension" : {

        "ID" : "contentgroup",

        "name" : "Content Group",

        "type" : "data",

        "Range" : null,

        "Properties" : null,

        "SubDimension" : {

            "ID" : "xIfaV44n7l5",

            "name" : "Content Sub-Group",

            "type" : "data",

            "Range" : null,

            "Properties" : null,

            "SubDimension" : null

        }

    },

 

Note that each sub dimension is listed as a hierarchical item of the dimension above it.

 

The key set of data that we want out of this result data is the list of measures.  The measures are all in a third segment bounded by square brackets – “[  ]”.  Each individual measure is defined within its own space bounded by curly brackets – “{  }”.  For each measure there are a number of definition items including, "name", "accumulationType", "ID","columnID", "measureFormatType", and "AllowTotals".  For purposes of our application we only need two of these values: “name” and “ID”.  The name filed is used for display to the user in the interface and the ID is used when we create the REST URL to request our data.  In the application we store this data in a custom data type.  This data type is shown below.

 

 
Public Type MEASUREDATA
    name As String
    id As String
    accumulationType As String
    columnID As String
    measureFormatType As String
    allowTotals As String
End Type
Public g_Measures() As MEASUREDATA
 

The function below is used to extract the data we need from the JSON result.  Once extracted the list of measures is available for the user to select. A check is also made to either enable or disable the use of the “search” functionality based on this being a single or multi-dimensional report.

 

Public Sub GetReportMeta()

   

    ' Get Report Meta so we can have list of measures

    Dim m_intWhere As Integer

    Dim m_strIsHier As String

   

    frmProfileList.lstMeasures.Clear                       ' Clear form we will display list on

    g_intMeasureCnt = 0                                    ' Initialize the profile count to 0

       

    ' Check to see if the user has selected the Profile Summary

    If frmProfileList.lstReports.ListIndex = frmProfileList.lstReports.ListCount - 1 Then

        frmProfileList.optMultiple.Enabled = False

        frmProfileList.optSingle.Enabled = False

        frmProfileList.optSEPeriods.Value = True

        Exit Sub                ' Exit this sub because there are no choices for measures

    Else

        frmProfileList.optMultiple.Enabled = True

        frmProfileList.optSingle.Enabled = True

        frmProfileList.optSEPeriods.Enabled = True

    End If

   

    ' Set up URL for fetching Report Meta

    g_strURL = g_strBaseREST & g_strSelectedProfileID & g_strReportsREST & g_Reports(frmProfileList.lstReports.ListIndex).id & g_strReportsMetaREST

   

    ' Call function to execute REST URL

    If Not RunREST(g_strURL, g_strUserName, g_strPassword) Then

        Exit Sub

    End If

       

    ' Check to see if the report is hierarchical - no search on those

    m_intWhere = InStr(1, g_strResult, "isHierarchy")                ' Find start point for measures list

    m_strIsHier = Mid(g_strResult, m_intWhere + 13, 4)

    If m_strIsHier = "true" Then

        frmProfileList.txtSearch.Enabled = False            ' disable text entry in search box

    Else

        frmProfileList.txtSearch.Enabled = True             ' enable text entry in search box

    End If

   

    ' Display the JSON URL Result

    m_intWhere = InStr(1, g_strResult, "[")                ' Find start point for measures list

    g_strResulta = Mid(g_strResult, m_intWhere + 1, Len(g_strResult) - (m_intWhere + 2))

   

    g_blnMore = True

    Do While g_blnMore

        g_intSplit = InStr(1, g_strResulta, "},")                 ' See if there is more than one entry

        If g_intSplit = 0 Then

            g_strResultb = Mid(g_strResulta, 2, Len(g_strResulta) - 2)

        Else

            g_strResultb = Mid(g_strResulta, 2, g_intSplit - 2)   ' Grab the first set of data

        End If

        g_strResultb = Replace(g_strResultb, """", "")            ' Get rid of the extra " characters

        g_strArr = Split(g_strResultb, ",")                       ' Create an array with this set of data

        ReDim Preserve g_Measures(g_intMeasureCnt)                ' Make room for new measure

        g_Measures(g_intMeasureCnt).name = Mid(g_strArr(0), 6)           ' Store the measure name

        frmProfileList.lstMeasures.AddItem g_Measures(g_intMeasureCnt).name    ' Add name for selection from form

        g_Measures(g_intMeasureCnt).id = Mid(g_strArr(2), 4)             ' Store the Measure ID

        g_Measures(g_intMeasureCnt).accumulationType = Mid(g_strArr(1), 18) ' Store the Accumulation Type

        g_Measures(g_intMeasureCnt).allowTotals = Mid(g_strArr(5), 13) ' Store the Allow Totals value

        g_Measures(g_intMeasureCnt).columnID = Mid(g_strArr(3), 10) ' Store the Column ID

               

       

        ' Check to see if we are done parsing

        If g_intSplit = 0 Then

            g_blnMore = False                             ' We're done

        End If

        g_strResulta = Right(g_strResulta, Len(g_strResulta) - g_intSplit - 1)   ' Set up for next profile

        g_intMeasureCnt = g_intMeasureCnt + 1               ' Bump the profile count

    Loop

   

    Application.Cursor = xlNorthwestArrow           ' Set the application cursor back to normal (arrow)

   

End Sub

 

So, now we have all of our data available in the user interface for the user to select what data will be included in the requested report.  The three keys areas of selection are the time periods, the search data (if available), and the selected measures.  Other selections include limiting what rows to return and how to deal with report totals.  I won’t go into all of the logic on these selections and the interactions, but there are certain checks that are made to ensure consistency and validity of the selected items.  But once these selections are made the user can create the URL and then copy it to the worksheet for eventual execution.

 

That’s all for this post.  The next and last post on this application will deal with processing the results of the report data requests.

 

Michael Love

Senior Solutions Engineer

0 Comments Permalink

This second post on dissecting the Sample Application deals with the process of requesting two separate, but related, sets of report data: the List of Available Reports for a Profile and the List of Report Periods for which data may be available.  I say “may be available” because it is possible that a report may not have existed when the Profile initially began analyzing.  A custom report may have been added some time after that initial set up process.  In that case the report would not have any data until when it had been enabled for the profile and the report template(s) available to the user.  This point really has nothing to do with the application.  I provided this comment only as a point of report data clarification.

 

The process in place within the application is that these two data requests will take place when the user clicks on a Profile Name in the combo list.  The request for Report List is made first followed by the List of Periods.  There is nothing that requires that one be done before the other; it’s just the sequence in which I chose to do them.  The REST URL to request a Report List looks like the following:

 

https://ws.webtrends.com/v1_1/ReportService/profiles/YcL4a5dufF6/reports/?format=json

 

Note that again I use the JSON data format to keep down the amount of data returned.  The process used to create this URL and send it is much like that used for the Profile List.  The difference, of course, is in the format of the REST URL where we need to include the Profile ID and then add the fields to identify the request as one for the Report List.

 

So, the data format that I get back looks very similar to the format used for the Profile List.

 

[

{"accountID":18079,

"profileID":null,

"name":"Creatives",

"ID":"WcTGeY1bNj5",

"language":null,

"type":null,

"Category":"Marketing",

"IsHierarchy":false,

"IntervalsEnabled":false,

"IsRealtimeCompatible":false,

"properties":null

},

{

"accountID":18079,

"profileID":null,

"name":"Creative Types",

"ID":"uZK6QjyBNj5",

"language":null,

"type":null,

"Category":"Marketing",

"IsHierarchy":false,

"IntervalsEnabled":false,

"IsRealtimeCompatible":false,

"properties":null

}

]

 

The formatting here has been added only to make reading the data easier.  The JSON format returns one long string bounded by square brackets – “[  ]”.  Within these brackets each report is detailed within a set of curly brackets – “{  }”.  Within those brackets are the individual meta items that define the report in a general format.  The two items that are really important to us here are the “name” and the “ID”.  The application parses this data using the following code.

 

' Display the JSON URL Result

    ' We need to parse out the results to get the data we need

    frmProfileList.lstReports.Clear                  ' Clear form we will display list on

    g_intReportCnt = 0                               ' Initialize the profile count to 0

    g_strResulta = Mid(g_strResult, 2, Len(g_strResult) - 2)       ' Strip the [] characters from each end

    g_blnMore = True

    Do While g_blnMore

        g_intSplit = InStr(1, g_strResulta, "},")    ' See if there is more than one entry

        If g_intSplit = 0 Then

            g_strResultb = Mid(g_strResulta, 2, Len(g_strResulta) - 2)  ' Grab the enry when only one left

        Else

            g_strResultb = Mid(g_strResulta, 2, g_intSplit - 2)     ' Grab the first set of data

        End If

        g_strResultb = Replace(g_strResultb, """", "")    ' Get rid of the extra " characters

        g_strArr = Split(g_strResultb, ",")               ' Create an array with this set of data

        ReDim Preserve g_Reports(g_intReportCnt)

        g_Reports(g_intReportCnt).id = Mid(g_strArr(3), 4)                      ' Store the Report ID

        g_Reports(g_intReportCnt).name = Mid(g_strArr(2), 6)                    ' Store the Report Name

        g_Reports(g_intReportCnt).accountid = Mid(g_strArr(0), 11)              ' Store the Account ID

        g_Reports(g_intReportCnt).profileid = Mid(g_strArr(1), 11)              ' Store the Profile ID

        g_Reports(g_intReportCnt).language = Mid(g_strArr(4), 10)               ' Store the Language

        g_Reports(g_intReportCnt).type = Mid(g_strArr(5), 6)                    ' Store the Report Type

        g_Reports(g_intReportCnt).category = Mid(g_strArr(6), 10)               ' Store the Report Category

        g_Reports(g_intReportCnt).ishierarchy = Mid(g_strArr(7), 13)            ' Store Hierarchy vlaue

        g_Reports(g_intReportCnt).intervalsenabled = Mid(g_strArr(8), 18)       ' Store Interval enablement

        g_Reports(g_intReportCnt).isrealtimecompatible = Mid(g_strArr(9), 22)   ' Store Realtime

        g_Reports(g_intReportCnt).properties = Mid(g_strArr(10), 12)            ' Store Properties

       

        ' Check to see if we are done parsing

        If g_intSplit = 0 Then

            g_blnMore = False                             ' We're done

        End If

        g_strResulta = Right(g_strResulta, Len(g_strResulta) - g_intSplit - 1)   ' Set up for next report

        g_intReportCnt = g_intReportCnt + 1               ' Bump the report count

    Loop

   

    ' Sort the report list arrays

    SortReports

   

    ' Load Report Names into form

    For i = LBound(g_Reports) To UBound(g_Reports)

        frmProfileList.lstReports.AddItem g_Reports(i).name     ' Add name for selection from form

    Next

   

The parsing process here is almost identical to what was done for the List of Profiles, the exception being the actual fields of data that are extracted and stored.  To store the data I have again set up a specific data type for the report data.  It looks like the following:

 

Public Type REPORTDATA

    accountid As String

    profileid As String

    name As String

    id As String

    language As String

    type As String

    category As String

    ishierarchy As String

    intervalsenabled As String

    isrealtimecompatible As String

    properties As String

End Type

Public g_Reports() As REPORTDATA

 

While this application does not yet use this, the “intervalsenabled” value is used to determine whether or not the trends option for report is valid.

Prior to loading the report names into the combo list on the form the application will sort that list into alphabetical sequence.  Unfortunately, the list returned from the request does not do that for you.  Using the custom data type here makes that sorting process easier because all of the report information I kept together.

 

The next step in the process here is to get the List of Periods that are available.  This is yet another REST URL format for this set of data.  This REST URL has the following format:

 

https://ws.webtrends.com/v1_1/ReportService/profiles/YcL4a5dufF6/periods/?format=json

 

This REST URL is almost identical to the Report List request with the one exception that it has the “periods” data request defined.  As before, we still use the JSON format for our data type.  This request is submitted just like all of the others we have done.  The format of the result data is shown in the following:

 

{"Report":["2009m01d04","2009m01d05","2009m01d06","2009m01d07","2009m01d08","2009m01d09","2009m01d10","2009w02","2009m01d11","2009m01d12","2009m01d13","2009m01d14","2009m01d15","2009m01d16","2009m01d17","2009w03","2009m01","2009q01","2009"],"Realtime":[“2009m01d18”]}

 

This format is a bit different than what we have seen with Profiles and Reports.  The outside delimiters in this case are the curly brackets – “{  }” – and the inside delimiters are the square brackets – “[  ]”.  In parsing this data all we care about are the individual periods.  The application, at this time also does not deal with the “Realtime” or express data.  That data will be ignored in this version of the application.  So, all we need to parse are the comma-delimited values, without the quote marks that are contained inside the first set of square brackets.  The code to do that looks like the following:

 

' Parse out the URL Result - set into individual time periods

    If InStr(1, g_strResult, "Realtime") Then

        g_strResultb = Mid(g_strResult, 12, (InStr(1, g_strResult, "Realtime") - 15)) ' Need to account for profiles with Express Data

    Else

        g_strResultb = Mid(g_strResult, 12, Len(g_strResult) - 29)    ' No express data

    End If

    g_strResultb = Replace(g_strResultb, """", "")            ' Get rid of the extra " characters

    g_strPeriods() = Split(g_strResultb, ",")                 ' Create an array with this set of data

       

    ' Load data onto Report Periods Form

    frmProfileList.lstPeriods.Clear                       ' Start with empty list

    For i = 0 To UBound(g_strPeriods)

        ReDim Preserve g_strReportPeriod(i)

        frmProfileList.lstPeriods.AddItem g_strPeriods(i)   ' Save in the form

        g_strReportPeriod(frmProfileList.lstPeriods.ListCount - 1) = g_strPeriods(i)    ' Save in our array for use later

    Next

 

This is fairly straightforward code to do this.  However, there are a few other items that we need to add to the list for the user to pick for time periods.  These are the relative starting point of “current day”, “current month”, and “current year”.  So these are manually added to our pick list since they have no representation in the list of periods returned.  The code to do that is as follow:

 

' Add relative time periods at end of form and array

    ReDim Preserve g_strReportPeriod(i + 3)

    frmProfileList.lstPeriods.AddItem "Today"

    g_strReportPeriod(frmProfileList.lstPeriods.ListCount - 1) = "current_day"

    frmProfileList.lstPeriods.AddItem "This Month"

    g_strReportPeriod(frmProfileList.lstPeriods.ListCount - 1) = "current_month"

    frmProfileList.lstPeriods.AddItem "This Year"

    g_strReportPeriod(frmProfileList.lstPeriods.ListCount - 1) = "current_year"

 

So, that ends this portion of the blog post.  In the next blog post we will be getting additional report meta information and dealing with creating a REST URL to fetch actual report data.

 

So, until the next posting,

 

Michael Love

Senior Solutions Engineer

1 Comments Permalink

It has often been said that Excel is the one application most often used when analyzing data on a computer.  Its use far outnumbers any other tool including those tools that fall within the category of Business Intelligence.  It is also our experience at Webtrends that our customers 8-25-2009 8-57-42 AM.pngalso use Excel to a great extent for the further analysis, integration , and presentation of data extracted or exported from Webtrends Analytics reports. So, it is important that Webtrends make available to its customers the tools needed to get that data from the Webtrends Analytics reporting tool and into Excel in an easy and straightforward way.  This is what the new REST URL DX API is all about.

 

There are multiple ways that you get data into Excel using the REST URL APIs.  The most important part of the process is to create/generate a valid REST URL in the first place.  Once that is done, it a fairly straightforward process to use it to get web data and populate rows of data where you want them.  Two very simple ways to do this, of course, are: 1) use the REST URL Generator application available at https://generator.webtrends.com; and 2) to use REST URLs that are generated with the new Webtrends Analytics 9 Insight interface (http://insight.webtrends.com).  In each of these all you need to know is your log in information and what data (report) you want to extract.  Using either of these methods will get you well on your way to populating Excel workbooks and then using that information in the way you want.

 

But, suppose you want to do more.  Suppose I want to extract data from Webtrends and then use the power to of Excel to store it, update it, and present it in the ways you need.  And, I want all of this in a single tool.  I don't want to go back and forth between a Webtrends application and Excel.  I want it all in Excel.

 

Well, this is possible.  Using tools such as VBA (Visual Basic for Applications) within Excel, you can access the Webtrends data, build REST URLs and populate the data into worksheets where you need the data.  The interesting part of this process is really how to you build and manage the REST URLs that extract your data.  If you read the developer notes on this site about how to build REST URLs, you will quickly learn that there is more than one format based on the data you need to get and that data can be returned in different formats.  These formats go by the names of XML, JSON, and HTML.  Somewhere imbedded in these formats hides the data I want and need to get the job done.  And why do you need to know this?  Because the various REST URL methods are needed to find out what data is available, such as profile, reports, time periods and the like. And you will need this information if you are going to build  a working REST URL that ultimately gets report data.

 

So, we have developed this Excel/VBA Sample Application that will make this easy for you.  The application has embedded a REST URL Generator functionally equivalent to the REST URL Generator mentioned above.  The VBA code that is part of the application uses all of the appropriate methods for extracting configuration information and parses it out from the data streams that come from using the REST URLs methods for doing things like getting a list of profiles and so forth.  The application also stores the REST URLs that you generate and lets you define where you want this data to go.

 

So, what does this mean for you?  Well, you can use the application as is.  It is quite functional and has complete usage documentation associated with it.  Or, as it is more intended, you can use this as a base for exploration or expansion on its functionality.  Add to it to do more formatting, analysis, data comparisons and the like.  Then feed what you did back into this developer network so that others may use your experience as a launching pad for their own application.

 

Whatever you do with it, enjoy!

 

Michael Love

Senior Solutions Engineer

2 Comments Permalink

WT.gifWe are pleased to announce the general availability of our Webtrends Data Extraction API.  This v1 release is the culmination of a ton of effort of many individuals inside Webtrends, as well as the terrific support from our partners and customers who have provided feedback and suggestions during our beta process.  Thank you all for helping us reach this milestone.

 

This open, innovative API service, provides you with elegant access to your powerful Webtrends analytics data.  This is the first time we have leveraged a full beta cycle as part of our iterative development process.  It is also the first time we've reached out to our community, via this Developer Network, to solicit feedback and validate our approach.  It's been a huge win for us.

 

What's new?

Recently, we've been working on some behind the scenes governance (security and usage controls), and focusing on fixing bugs/issues.  We've also modified the HTML and CSV formats slightly to return data back in a more easy-to-consume structure, per your suggestions.  These changes, along with some additional functionality has made this API much easier to use directly within Excel, making it the only API in the industry that is powerful enough for sophisticated applications, while easy enough to also be used by everyone via Excel.

 

We've launched the formal "/v1/" version structure into the URL.  If you are using the "/beta/" request in your URL, you should switch over to "/v1/".  We'll continue to use the "/beta/" structure for future updates, so please don't use it for any production services or applications.  Please consult our documentation for the most current information on the API.

 

What's next?

We are going to continue to iterate rapidly on this Data Extraction platform.  It's already become very popular with our customers and partners as it is so easy to use.  Look for a regular maintenance release on a monthly basis, along with a 3-4 month cycle of releases that will include awesome new features.

 

We will continue to use this Developer Network for communications, ideas, and general sharing of information.  As always, please feel free to login and post questions, respond to other questions, or post ideas you have.  Your creativity and innovation is contagious.  It's all good.

0 Comments Permalink

I did a blog entry a while back in regards to doing asynchronous web requests against REST based web services, which is what the Webtrends Web Services are built against.  This entry I'll show how to setup a synchronous web request for data against the services.

 

Benifits:

  • The request is made and in a top down fashion the response is returned with the respective string or stream object data needed.
  • Easier to test versus asynchronous web requests.

 

Disadvantages:

  • The request is made, and will pause or "freeze" while waiting for the response.  In a Windows Forms, WPF, Silverlight (or Flash, etc) client application this will cause the user interface to become unresponsive while waiting for the response.

 

So how do we make a synchronous call?  First create a HttpWebRequest object and instantiate it.

 

            var req = (HttpWebRequest)WebRequest.Create(@"https://ws.webtrends.com/beta/ReportService/profiles/?format=json");

 

Next you'll want to setup credentials.

 

            req.Credentials = new NetworkCredential(@"YourWebTrendsAccount\YouUserName", "YourPassword");

Now make sure when the request is made request the server to use compression.  This is entirely transparent and only needed at this single stage of the request.

 

            req.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");

 

Finally set the method of request and other peripheral properties.

 

            req.Method = "Get";

            req.AllowAutoRedirect = true;

 

Once this is done you're ready to submit the request.

 

                WebResponse res = req.GetResponse();

 

To retrieve a simple string object feed through to the end of a the stream object response that is available via the WebResponse object.  This can also prospectively be done with the actualy HttpWebRequest object as it has a GetResponseStream() method.  I've segmented it here for example.

 

                string jsonResponse = (new StreamReader(res.GetResponseStream()).ReadToEnd());

 

The following is the complete code with exception handling.

 

            // Create a request

            var req = (HttpWebRequest)WebRequest.Create(@"https://ws.webtrends.com/beta/ReportService/profiles/?format=json");


            req.Credentials = new NetworkCredential(@"YourWebTrendsAccount\YouUserName", "YourPassword");

            // Add compression request.  IIS will return a compressed data result.

            req.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");

            req.Method = "Get";

            req.AllowAutoRedirect = true;


            // Read the response

            try

            {

                WebResponse res = req.GetResponse();

                string jsonResponse = (new StreamReader(res.GetResponseStream()).ReadToEnd());


            }

            catch (WebException exw)

            {

                WebResponse response = exw.Response;

                string error = (new StreamReader(response.GetResponseStream())).ReadToEnd();

            }

 

Hope that helps!  Leave a comment or question if anything comes up while coding!

 

In a following entry the team will be providing more examples in consumption of these results programmaticaly, but as mentioned by Developer_Donut's entry we'll be releasing a software SDK soon.  This SDK will drammatically simplify and provide guidance on how to access and to utilize the web services.

0 Comments Permalink

When making requests for reports via the WebTrends Web Services the best practice is to get the data compressed.  When making an HTTP request it's really easy to make sure the web server will compress the data and speed up the transfer dramatically.  So how does one get the content type set to make a request for compressed results programmatically?  Easy, and here's a quick example I put together in C# with the standard HttpWebRequest objects.


        [Test]
        public void TestBaselineCompressedRequest()
        {
            var request = (HttpWebRequest)WebRequest.Create("https://somedomain/theservicepath/to/rest/?format=xml");
            request.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");
            request.Credentials = new NetworkCredential("someUser", "somePassword");

 

            var response = request.GetResponse();
            var responseStream = new StreamReader(response.GetResponseStream(), Encoding.UTF8);
            string getTheResults = responseStream.ReadToEnd();

 

            Assert.IsNotNull(responseStream);
            Assert.IsNotNull(getTheResults);

        }

 

This should get anyone kick started getting compressed data from WebTrends REST Web Services.

0 Comments 0 References Permalink

Today I'm going to cover material around connecting to WebTrends new REST based Web Services.  To start off, let's cover the context behind the methods of making calls to these services.

 

There are two specific ways in which to connect to REST based Web Services, asynchronously and synchronously.  Depending on the type of application, tool, or other mechanism you are connecting to the services with you will want to use one or the other or might be using one or the other.

 

When making an asynchronous call against a service the application, or to be specific the calling thread, will initiate the call but continue executing without waiting for a response.  When the response is received an event fires and the data returned can then be handled in some way.  For client service applications, server based applications, or anything that needs to return control to the application or UI thread, this is the method to use.  This provides a smoother and more continuous application flow when an end user is accessing services.

 

A synchronous call against a service executes on the actual application thread itself, and holding that thread until a response returns.  This can make a UI appear to hang or freeze.  Often this is only used if there is no way to return data via a post event.  One scenario that would need to do this would be to render a page, or display, in its entirety before returning it to the user.  This is a common scenario on the web.  AJAX solves this to some degree, but often one still needs to render the entire initial page, and thus, must wait for the data to return.

 

Best Practice

 

The best practice is to use asynchronous calls.  Only use synchronous calls for stand alone processing or non-user viewed feeds.

 

A Simple Example

 

Here is a C# asynchronous call being made against our WebTrends Web Services.  Keep in mind, I'm providing these as examples, I do NOT suggest writing tests with Thread.Wait calls in them.

 

The example also shows how to setup your WebTrends Credentials for authenticating to our WebTrends Web Services.

 

    [TestClass]
    public class GetRestServices
    {
        private XDocument doc;

 

        private void svc_DownloadStringCompleted(object sender, DownloadStringCompletedEventArgs e)
        {
            doc = XDocument.Parse(e.Result);
        }

 

        [TestMethod]
        public void TestAsynchronousCallWithSecurity()
        {
            const string baseUri = "https://ws.webtrends.com/beta/reportservice/profile/?format=xml";

 

            var svc = new WebClient();
            svc.Credentials = new NetworkCredential("yourWebTrendsAccount\WebTrendsUserName", "yourSuperSecretPassword");
            svc.DownloadStringCompleted += svc_DownloadStringCompleted;
            svc.DownloadStringAsync(new Uri(baseUri));

 

            Thread.Sleep(3000);
            Assert.IsNotNull(doc);
            var docTest = new XDocument();
            Assert.IsInstanceOfType(doc, docTest.GetType());
        }
    }

 

Hope those are helpful. 

0 Comments 0 References Permalink

Web Services & REST

Posted by Adron Hall Mar 11, 2009

This is my hello world blog post for the Developer Network.  Over time I will be posting many blog posts, primarily focused on development of and around the Web Services & using REST Web Servies in general.  One of the first topics I'll cover is using authenticating programmatically to the WebTrends Web Services.  I'll follow up that post in short order with more REST specific and data specific pieces.  The primary language I'll be using for examples is C#, from different perspectives in the .NET Platform.  With this post though, I'm not going to touch on the services technical aspects, but instead on the context and reason for their existence.

 

REST Services in the Web

 

REST stands for Representational State Transfer.  REST services are really more than mere Web Services, as inherent to the acronym.  REST started as a disertation by Roy Fielding as pointed out in the Wikipedia Article on REST Services.  Give that a read, as I'm going to skip ahead and cover some of the other context points of REST.

 

Even with the massive push from enterprises (and a few large software companies) for the SOAP Web Services, the REST Web Services have really taken over the web.  A lot of this has to do with the simple, straight forward, and less overhead involved with REST based transfers.  As the web has grown, SOAP became less viable and REST increased in viability.  In this course of events we decided to go with a REST based services offering.

 

One of the main cocepts around REST is that of resources.  Almost everything with the REST ideal is embodied around a global identifier, which is a URI, pointing to a particular resource.  These resources are then manipulated by standard HTTP interfaces.  In our case, we provide primarily resources at this point in read only XML, JSON, and Excel friendly XML, without manipulation.  In the future we'll exand to provide other capabilities to manipulate these resources.

 

Some of the key elements of REST have enabled us to move quickly and efficiently to build out the needed infrastructure.  In addition several of the key elements enable even further scalability and capabilities.  Some of these are the abstraction of application state and functionality into resources.  This absolves the application developer of the complexity of SOAP Web Services while increasing server response time & load.  Another great benifit of the REST approach is that it depends less on vendor software and specific technology stacks, in other words .NET, Java, PHP, Ruby on Rails, Adobe Flex or anything else can be used to connect without needing or utilizing any dependent stack components.

 

What I've covered here are a few quick points on REST services, links for more information, and a few basic reasons why we've chosen REST for these service offerings.  Keep reading and we'll soon have more information on authenticating, retrieveing, and other programmatic use of the WebTrends Web Services.

0 Comments 0 References Permalink

In order to leverage the new Webtrends REST web services, you need to configure a user in your Webtrends account properly.  We describe this in our Getting Started document.  I wanted to add a little more information for your reference.

 

Any user can be setup to use the Web Services functionality.  If you are using the Web Services for pulling data into Excel for a static or one-time report, it's fine to use your existing user account.

 

However, if you are planning on leveraging the Web Services data on a regular basis, or via an automated program or application, you'll be much better off by creating a new user account for that purpose.

 

For example, you might define a new user called "WSUser" that you only plan on using for pulling Web Services data.  This is the best practice we've adopted within WebTrends for our internal consumption of WebTrends data.

 

Questions?  Comments?

0 Comments 0 References Permalink