CEP session agent schema

The CEP session agent uses a schema to deliver messages to the receiving systems for FILE, TCP (Delimited and XML format), and Coral8 output.

The receiving systems require the data to be organized in a format that can be consumed easily. The raw request text data is in name/value format and needs to be converted into a record format.

Schema output considerations

The schema for delivering messages for TCP (SWA - SAS Web Analytics format) output is different. The schema has only Page level information and no Events. For example:
IP
<blank>
<blank>
PageTimeStamp
RequestLine
StatusCode
ResponseType
Referrer
UserAgent
Cookies
<blank>
ServerIP
<blank>
RequestMethod
ServerPort
ConnectionID
<blank>
QueryString
WSGenerationTime
<blank>
TLTSID
TLTUID
RequestSize

Finally, HTTP output uses the configurable schema in the CEPCustomFields.cfg file under the [HTTPFields] section.

The following table presents the schema is used by the CEP session agent to deliver messages to receiving systems. There are 33 fixed values and 100 custom fields.

Table 1. Default Event Bus data schema
POS Name Discover REQ Field Type Length Description

1

Canister Canister

String

64

Name of server for the session

2

TLTSID TLTSID

String

32

Sessionization string used by Discover (typically a 32 character GUID)

3

 TLTHID TLTHID

String

32

Hit ID (32 Character GUID)

4

 TLTUID TLTUID

String

32

User ID (usually a cookie)

5

 SessionID SesnIdx

Long

Number assigned to session for the in memory database (64 bit integer)

6

 HitNumber HitNumber

Long

The page number in the session as observed by Discover

7

 EventID TLEventID

Integer

Event ID

8

 FactID TLFID

Integer

Fact ID

9

 ReportGroupID TLDimGroupID

Integer

Report Group ID (Dimension Group ID)

10

 FactValue TLFactValue

String

256

Fact Value (text found)

11

 FactNumericValue TLFactNumericValue

Num

Numeric value of fact value (if applicable, 0.0 default value)

12

 DimValue1 TLDim1

String

256

Dimension value 1 text (TLT$NULL if empty)

13

 DimValue2 TLDim2

String

256

Dimension value 2 text (TLT$NULL if empty)

14

 DimValue3 TLDim3

String

256

Dimension value 4 text (TLT$NULL if empty)

15

 DimValue4 TLDim4

String

256

Dimension value 4 text (TLT$NULL if empty)

16

 PageTimeTicks RequestTimeEx

Long

Time of page in milliseconds in unix time (GMT, number of milliseconds since 1970)

17

 IP REMOTE_ADDR

String

16

The IP address for the session

18

 IPNumber

Long

IP address (REMOTE_ADDR ) that is translated and converted to an integer (for easy lookup against GEO IP database).

For example, if the IP address is w.x.y.z, then the IP Number is translated as: 16777216*w + 65536*x + 256*y + z.

19

 URL URL

String

256

URL for page excluding the query string

20

 TLT_URL TLT_URL

String

256

The normalized Discover URL for reporting

21

 QueryString HTTP_QUERY

String

1024

The query string for the URL

22

 Referrer HTTP_REFERER

String

1024

The referrer for the page

23

 UserAgent HTTP_USER_AGENT

String

256

The browser for the page

24

 RequestMethod REQUEST_METHOD

String

16

The Request Method (GET/POST)

25

 StatusCode StatusCode

Integer

HTTP Status Code for page

26

 ReqCancelled ReqCancelled

Boolean

Request Cancelled flag from client or server (True/False)

27

 ReqDiscarded ReqDiscarded

Boolean

Request Discarded flag from client or server (True/False)

28

 RequestSize RequestSize

Long

Size in bytes of the HTTP Request

29

 ResponseSize ResponseSize

Long

Size in bytes of the HTTP Response

30

 WSGenerationTime WS_Generation

Long

Generation time for the page on the server in microseconds

31

 NTTotalTime NT_Total

Long

Network time for the page in microseconds (last ACK by client)

32

 RTTotalTime RT_Total

Long

Total time for page in microseconds

33

 ConnSpeed ConnSpeed

Long

Connection speed in bits/second

34/53

 CustomField0 Custom Field String 256

The custom fields can be set using CEPCustomFields.cfg. See Custom fields for a description of the configuration sections in CEPCustomFields.cfg.

Event IDs are listed in the tooltip for an event in the Discover Event Manager. See the Unica Discover Event Manager Manual for information on working in the Manage Events - Events Tab.

Fact data, including dimension values, is available only through the request, which can be viewed through Replay. For information on Replay, see the Unica Discover Administration Manual.

Custom fields

The custom fields are populated with the configuration in the CEPCustomFields.cfg file, which is located in the Discover installation directory.
Note: The CEPCustomFields.cfg file should be installed in the Discover install directory on the server where Data Export for Data Analysis is installed.
The CEP session agent looks for changes in CEPCustomFields.cfg  and automatically updates the rules for processing the events without requiring a restart of the pipeline.
Note: The changes may take up to one minute to update in the CEP session agent.

There are 100 custom fields in each record. The custom fields can be populated with any value from the request.

The custom fields cannot be populated with data from the response during the event processing. If you require data from the response, you must create custom request variables in the main pipeline with the RTA or PrivacyEx session agent(s).

The CEPCustomFields.cfg file is used for the following configuration:
  • Event Include
  • Event Omit
  • Default Event Custom Fields
  • Specific Event Custom Fields
  • HTTP Message Configuration

The following table provides a description of each section.

Table 2. Sections in CEPCustomFields.cfg 
Configuration file section Description
[EventInclude]

This section can be used to only send events with the correct EventID (UniqueID in the CEP section). This value overrides any event ID's set in the [EventOmit] section.
[EventOmit]

This section is used to remove events from the event bus data stream. Any event in the EventID list will not be sent to the downstream system.
[DefaultEvent]

Every event bus hit that does not have a specific event entry in the configuration file inherits the custom field settings and status code filters specified in this section.
[EventX] 

(where X is a sequential number)

Each section contains specific custom field configuration for a specific event ID, which is helpful when sending data in form fields only present under certain conditions (i.e. new account sign up form).
[DefaultPage]

Every hit that does not have a specific page entry in the configuration file inherits the custom field settings and status code filters specified in this section.
[PageX]

(where X s a sequential number)

Each section contains specific custom field configuration for a specific page identifier, as indicated by the reqField and Filter values.

Sample CEP output

The following is a sample of the CEP file output.

In order to fit the sample on the page, it has been broken into sections.
Note: The actual CEP output includes data only (it does not include headers). The headings are included in the samples for illustrative purposes only.
Figure 1. Sample data for Canister, TLTSID, and TLTHID output fields
Canister         TLTSID                            TLTHID
ADMINIB-0MV8KCT  2A8F2E159C798E2868179C711F49BEC2  C5DC226E31C41031006BED4DEA3ED43
ADMINIB-0MV8KCT  2A8F2E159C798E2868179C711F49BEC2  C5DC226E31C41031006BED4DEA3ED43
ADMINIB-0MV8KCT  2A8F2E159C798E2868179C711F49BEC2  C5DC226E31C41031006BED4DEA3ED43
ADMINIB-0MV8KCT  2A8F2E159C798E2868179C711F49BEC2  C5DC226E31C41031006BED4DEA3ED43
ADMINIB-0MV8KCT  2A8F2E159C798E2868179C711F49BEC2  C5DC226E31C41031006BED4DEA3ED43

See Default Event Bus data schema for a description of these fields.

Figure 2. Sample data for TLTUID, SessionId, HitNumber, EventID, FactID, ReportGroupID, and FactValue output fields
TLTUID   SessionId   HitNumber   EventID   FactID   ReportGroupID   FactValue
               213          48         0      380               0   § %5Cn   CompXYZ.co.uk Sign In%5Cn
               213          48         0      415               0   § %5Cn   CompXYZ.co.uk Sign In%5Cn
               213          48         0      416               0   § %5Cn   CompXYZ.co.uk Sign In%5Cn
               213          48         0      552               0   Hans-Jorgen
               213          48         0      580               0   /ap/signin

See Default Event Bus data schema for a description of these fields.

Figure 3. Sample data for FactNumericValue, DimValue1, DimValue2, DimValue3, DimValue4, and PageTimeStamp output fields
FactNumericValue   DimValue1    DimValue2    DimValue3    DimValue4    PageTimeStamp
               0   TLT$NULL     TLT$NULL     TLT$NULL     TLT$NULL     1410898825023
               0   BROWSER      TLT$NULL     TLT$NULL     TLT$NULL     1410898825023
               0   BROWSER      TLT$NULL     TLT$NULL     TLT$NULL     1410898825023
               0   TLT$NULL     TLT$NULL     TLT$NULL     TLT$NULL     1410898825023
               0   TLT$NULL     TLT$NULL     TLT$NULL     TLT$NULL     1410898825023

See Default Event Bus data schema for a description of these fields.

Figure 4. Sample data for IP, IPNumber, URL,and TLT_URL output fields
IP               IPNumber      URL          TLT_URL
192.168.76.102   3232255078    /ap/signin   /ap/signin
192.168.76.102   3232255078    /ap/signin   /ap/signin
192.168.76.102   3232255078    /ap/signin   /ap/signin
192.168.76.102   3232255078    /ap/signin   /ap/signin
192.168.76.102   3232255078    /ap/signin   /ap/signin

See Default Event Bus data schema for a description of these fields.

Figure 5. Sample data for QueryString output field
Note: Because of its length, the sample includes forced line-breaks.
QueryString
 _encoding=XXXX&openid.assoc_handle=XXXXXXXXXXXXXXXXXX&openid.claimed_
id=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.identity=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.mode=XXXXXXXXXXXXX&openid.ns=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.ns.pape=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.pape.max_auth_age=X&openid.return_
to=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&pageId=XXXXXXXXXXXXXXXXXX
&showRmrMe=X&siteState=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

 _encoding=XXXX&openid.assoc_handle=XXXXXXXXXXXXXXXXXX&openid.claimed_
id=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.identity=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.mode=XXXXXXXXXXXXX&openid.ns=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.ns.pape=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.pape.max_auth_age=X&openid.return_
to=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&pageId=XXXXXXXXXXXXXXXXXX
&showRmrMe=X&siteState=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

 _encoding=XXXX&openid.assoc_handle=XXXXXXXXXXXXXXXXXX&openid.claimed_
id=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.identity=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.mode=XXXXXXXXXXXXX&openid.ns=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.ns.pape=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.pape.max_auth_age=X&openid.return_
to=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&pageId=XXXXXXXXXXXXXXXXXX
&showRmrMe=X&siteState=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

 _encoding=XXXX&openid.assoc_handle=XXXXXXXXXXXXXXXXXX&openid.claimed_
id=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.identity=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.mode=XXXXXXXXXXXXX&openid.ns=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.ns.pape=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.pape.max_auth_age=X&openid.return_
to=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&pageId=XXXXXXXXXXXXXXXXXX
&showRmrMe=X&siteState=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

 _encoding=XXXX&openid.assoc_handle=XXXXXXXXXXXXXXXXXX&openid.claimed_
id=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.identity=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.mode=XXXXXXXXXXXXX&openid.ns=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.ns.pape=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
openid.pape.max_auth_age=X&openid.return_
to=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&pageId=XXXXXXXXXXXXXXXXXX
&showRmrMe=X&siteState=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

See Default Event Bus data schema for a description of this field.

Figure 6. Sample data for Referrer output field
Referrer
http://www.CompXYZ.co.uk/gp/huc/view.html?ie=XXXX&newItems=XXXXXXXXXXXXXXXXXX
http://www.CompXYZ.co.uk/gp/huc/view.html?ie=XXXX&newItems=XXXXXXXXXXXXXXXXXX
http://www.CompXYZ.co.uk/gp/huc/view.html?ie=XXXX&newItems=XXXXXXXXXXXXXXXXXX
http://www.CompXYZ.co.uk/gp/huc/view.html?ie=XXXX&newItems=XXXXXXXXXXXXXXXXXX
http://www.CompXYZ.co.uk/gp/huc/view.html?ie=XXXX&newItems=XXXXXXXXXXXXXXXXXX

See Default Event Bus data schema for a description of this field.

Figure 7. Sample data for UserAgent output field
UserAgent
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36

See Default Event Bus data schema for a description of this field.

Figure 8. Sample data for RequestMethod, StatusCode, ReqCancelled, ReqDiscarded, RequestSize, and ResponseSize output fields
RequestMethod    StatusCode    ReqCancelled    ReqDiscarded    RequestSize    ResponseSize
GET                    200     false           false                  1710           30885
GET                    200     false           false                  1710           30885
GET                    200     false           false                  1710           30885
GET                    200     false           false                  1710           30885
GET                    200     false           false                  1710           30885

See Default Event Bus data schema for a description of these fields.

Figure 9. Sample data for WSGenerationTime, NTTotalTime, RTTotalTime, ConnSpeed, and CustomField0 output fields
WSGenerationTime       NTTotalTime      RTTotalTime      ConnSpeed 
          177873               260           178133       331323076
          177873               260           178133       331323076
          177873               260           178133       331323076
          177873               260           178133       331323076
          177873               260           178133       331323076

See Default Event Bus data schema for a description of these fields.

Figure 10. Sample data for CustomField1,CustomField2, and CustomField3 output fields
CustomField0                     CustomField1    CustomField2    CustomField3
A8F2E159C798E2868179C711F49BEC2                  Win7            BROWSER
A8F2E159C798E2868179C711F49BEC2                  Win7            BROWSER
A8F2E159C798E2868179C711F49BEC2                  Win7            BROWSER
A8F2E159C798E2868179C711F49BEC2                  Win7            BROWSER
A8F2E159C798E2868179C711F49BEC2                  Win7            BROWSER

See Default Event Bus data schema for a description of custom fields.