Clean-up Journey Audiences

From V12.1.7 onwards, journey audiences can be removed from live running journeys. There will be a cron job scheduled (customer can configure cron expression on need basis), which will find all audiences with status JOURNEY_COMPLETED and JOURNEY_ENDED and remove those, as well as their related data (like audience responses, milestone, goals, etc data) from respective tables. This removed data will not be recoverable. This will help in improving overall performance of journey engine. To support this, we introduced below properties in engine's application.properties file.
  1. cleanup.audience.enable=true/false - Journey data clean-up is enabled/disabled respectively.
    Note: This setting is used for deleting audiences of deleted journey and its associated data from Journey system database tables.
  2. archival.audience.enable=true/false - Journey audiences archival from journeyaudiences to endjourneyaudience table is enabled/disabled respectively.
  3. archival.audience.ttl=X - If cleanup.audience.enable=true, Journey audiences older than X-days in Journey, then the Journey audiences and their associated data will get deleted at set frequency archival.audience.cron, Journey audiences and their associated data less than X-days older in Journey will remain available in Journey.
    Note: Default value of archival.audience.ttl is 365 days. User can set minimum 2 days, if user adds any value less than 2 days, then its equivalent to 2 days for this property.
  4. archival.audience.cron=0 0/5 * * * ? -> Journey audience clean up will be executed according to set frequency of this cron job.

    Example:

    cleanup.audience.enable=true

    archival.audience.enable=true

    archival.audience.ttl=3

    archival.audience.cron=0 0/5 * * * ?

    In above case, as archival.audience.enable=true, audiences with status Journey_completed/Journey_ended will be moved from journeyaudiences table to endjourneyaudience table and as user set cleanup.audience.enable=true and archival.audience.ttl=3, Journey audiences with Journey_completed or Journey_ended and older than set 3-days will get cleaned up along with their data available in the database tables at set frequency "archival.audience.cron=0 0/5 * * * ?"

If journey engine is deployed in cluster mode:
  1. To run both archival and cleanup process on one node, make both flags true on that one node.
  2. To disable both archival and cleanup processes, make them false on all nodes.
  3. To run archival and cleanup processes on different engine nodes, make these properties individually true on any two engine server nodes.

    Example: Let's consider, in cluster we have four engine nodes, if user want to run clean-up process on two different engine node then user needs to do the below settings:

    cleanup.audience.enable=true server 1

    archival.audience.enable=false server 1

    cleanup.audience.enable=false server 2

    archival.audience.enable=true  server 2

    cleanup.audience.enable=false server 3

    archival.audience.enable=false server 3

    cleanup.audience.enable=false server 4

    archival.audience.enable=false server 4

Note:
While deleting millions of Journey audiences and their related data, QueryTimeOut exception may occur . To avoid this, we are recommending to create below indexes immediately after installing or upgrading Journey to 12.1.7 or before starting Journey engine:
  • create index MY_UJR_IDX_LAF on LoopAudienceFlow(audienceId);
  • create index MY_UJR_IDX_BIDADM on BatchIDAudiencedataMap(audienceId);
  • create index MY_UJR_IDX_JAG on JourneyAudienceGoal(audienceId);
  • create index MY_UJR_IDX_ARHD on AudienceResponseHTTPDetail(audienceResponseId);
  • create index MY_UJR_IDX_ARE on AudienceResponseExtended(audienceResponseId);
  • create index MY_UJR_IDX_ARMD on AudienceResponseMetaData(audienceResponseId);
  • create index MY_UJR_IDX_ARI on AudienceResponseInteraction(audienceResponseId);
  • create index MY_UJR_IDX_AR on AudienceResponse(audienceId);
  • create index MY_UJR_IDX_JAF on JourneyAudienceFlow(audienceId);
  • create index MY_UJR_IDX_AWS on AudienceWaitState(audienceId);
  • create index MY_UJR_IDX_JDRM on JourneyDeliverResponseMaster(audienceId);
  • create index MY_UJR_IDX_JAM on JourneyAudienceMilestone(audienceId);