Clean-up Journey Audiences

From V12.1.7 onwards, journey audiences can be removed from live running journeys. There will be a cron job scheduled (customer can configure cron expression on need basis), which will find all audiences with status JOURNEY_COMPLETED and JOURNEY_ENDED and remove those, as well as their related data (like audience responses, milestone, goals, etc data) from respective tables. This removed data will not be recoverable. This will help in improving overall performance of journey engine. To support this, we introduced below properties in engine's application.properties file.

cleanup.audience.enable=true/false - Journey data clean-up is enabled/disabled respectively.
Note: This setting is used for deleting audiences of deleted journey and its associated data from Journey system database tables.
If cleanup.audience.enable=true, then below are the side effects on Journey audience counts:
1. Journey Stats count
  If journey that had audiences older than (archival.audience.ttl=x) days ago - the data for these audiences will be deleted and hence the count on journey start node will be decreased accordingly.
  As audience records are deleted, their respective responses (if any) will also get deleted, hence responses (visible on stats) available on touchpoints (Email, SMS, Push, WhatsApp, REST, JDBC, Loop, Salesforce) will also get decreased accordingly.
  Note:
  
  Publish touchpoint statistics for audiences will remain the same and is not impacted by above deletion.
  
  Adtech touchpoint statistics for audiences will remain the same and is not impacted by above deletion.
  
  Journey Audience Flow counts across journey canvas branches will not be affected.
  
  Audiences that came in journey in less than (archival.audience.ttl=x) days will not be affected and will remain as-is on journey statistics and start node.
2. Journey Goals and Goal Analysis
  - Journey that had goals achieved by audiences which are older than (archival.audience.ttl=x) days - the goal achieved count will get decreased accordingly after deletion.
  - For frequency and non-frequency based goals - As audience response are deleted, achieved target will also get decreased accordingly. Hence, the set target will have to be achieved again.
  - For non frequency based goals - If there is journey goal complete criteria marked for aspecific goal, the journey will be completed after achieving the set target.
  - On Journey Goal Analysis Page - The graphs for all goals will be displayed as per the new goal achieved count after deletion of journey audiences.
3. Journey Audience Analysis Tab
  - If journey that had audiences older than (archival.audience.ttl=x) days ago, the data for these audiences will be deleted and hence the count on journey audience analysis tab will be decreased, and graphs will be represented accordingly.
4. Milestones on journey canvas page
  After deletion of audiences, milestones count on journey canvas page will get refreshed only when new audiences comes in Journey milestone and then only user can see reduced count on milestone canvas page.
5. Milestone Analysis Tab
  If journey that had audiences older than (archival.audience.ttl=x) days ago - Count for milestoneanalysis page will get reduces same as Mileston canvas page. Hence, Milestone Achieved By (%) will be affected.
6. Journey Performance Tab
  After deletion of audiences , counts on performance tab for Email , SMS and Push Tiles will not have any impact and will remain as-is.
7. Loop
  If journey that had audiences older than (archival.audience.ttl=x) days ago - Response counts shown on loop details link will also be affected accordingly.
archival.audience.enable=true/false - Journey audiences archival from journeyaudiences to endjourneyaudience table is enabled/disabled respectively.
archival.audience.ttl=X - If cleanup.audience.enable=true, Journey audiences older than X-days in Journey, then the Journey audiences and their associated data will get deleted at set frequency archival.audience.cron, Journey audiences and their associated data less than X-days older in Journey will remain available in Journey.
Note: Default value of archival.audience.ttl is 365 days. User can set minimum 2 days, if user adds any value less than 2 days, then its equivalent to 2 days for this property.
archival.audience.cron=0 0/5 * * * ? -> Journey audience clean up will be executed according to set frequency of this cron job.
Example:
cleanup.audience.enable=true
archival.audience.enable=true
archival.audience.ttl=3
archival.audience.cron=0 0/5 * * * ?
In above case, as archival.audience.enable=true, audiences with status Journey_completed/Journey_ended will be moved from journeyaudiences table to endjourneyaudience table and as user set cleanup.audience.enable=true and archival.audience.ttl=3, Journey audiences with Journey_completed or Journey_ended and older than set 3-days will get cleaned up along with their data available in the database tables at set frequency "archival.audience.cron=0 0/5 * * * ?"

If journey engine is deployed in cluster mode:

To run both archival and cleanup process on one node, make both flags true on that one node.
To disable both archival and cleanup processes, make them false on all nodes.
To run archival and cleanup processes on different engine nodes, make these properties individually true on any two engine server nodes.
Example: Let's consider, in cluster we have four engine nodes, if user want to run clean-up process on two different engine node then user needs to do the below settings:
cleanup.audience.enable=true server 1
archival.audience.enable=false server 1
cleanup.audience.enable=false server 2
archival.audience.enable=true server 2
cleanup.audience.enable=false server 3
archival.audience.enable=false server 3
cleanup.audience.enable=false server 4
archival.audience.enable=false server 4

Note:

While deleting millions of Journey audiences and their related data, QueryTimeOut exception may occur . To avoid this, we are recommending to create below indexes immediately after installing or upgrading Journey to 12.1.7 or before starting Journey engine:

create index MY_UJR_IDX_LAF on LoopAudienceFlow(audienceId);
create index MY_UJR_IDX_BIDADM on BatchIDAudiencedataMap(audienceId);
create index MY_UJR_IDX_JAG on JourneyAudienceGoal(audienceId);
create index MY_UJR_IDX_ARHD on AudienceResponseHTTPDetail(audienceResponseId);
create index MY_UJR_IDX_ARE on AudienceResponseExtended(audienceResponseId);
create index MY_UJR_IDX_ARMD on AudienceResponseMetaData(audienceResponseId);
create index MY_UJR_IDX_ARI on AudienceResponseInteraction(audienceResponseId);
create index MY_UJR_IDX_AR on AudienceResponse(audienceId);
create index MY_UJR_IDX_JAF on JourneyAudienceFlow(audienceId);
create index MY_UJR_IDX_AWS on AudienceWaitState(audienceId);
create index MY_UJR_IDX_JDRM on JourneyDeliverResponseMaster(audienceId);
create index MY_UJR_IDX_JAM on JourneyAudienceMilestone(audienceId);