PSM Uploader and Encoder Flow

This article provides an in-depth overview of the PSM Uploader and Encoder services, focusing on their key functions and operational mechanisms. A crucial component of these services is the role of automated tasks, commonly known as CronJobs, which are instrumental for their efficient operation. The Encoder service is primarily responsible for converting video files into an upload-ready format, while the Uploader service takes care of transferring these processed files to a designated storage location. The aim of this article is to offer a comprehensive understanding of how these services operate – both independently and in conjunction – to ensure optimal performance within the PSM Uploader tool.

The below diagram depicts the flow:

Encoder Service

The Encoder service converts video files from .guac format to .mp4. This transformation is essential for preparing the files for uploading.

1.1 CronJob 1: Session Checks & Queueing

Running every 5 minutes, the CronJob reads filenames from a predetermined directory. The job filters out sessions based on their status, which it fetches from the Endpoint ID (EID) using a specific API call.

The workflow for CronJob 1 is as follows:

1. Reading Filenames: Reads filenames from a designated directory (e.g., /session_files/)./session_files/

2. Closed/Terminated Sessions: Use the EmpowerID API endpoint /api/services/v1/secureaccessgateway/polling to check for closed or terminated sessions.

3. Filtering Sessions: Filters out sessions with a null recording status.

4. Queueing: Adds the sessions to the sessionsQueue, after verifying they do not already exist in the queue or the failedJobs array.

1.2 CronJob 2: Processing & Retrying Failed Jobs

Activated every 10 minutes, this CronJob functions independently of the first. It primarily deals with processing queued sessions and managing failures.

The workflow for CronJob 2 is as follows:

1. Retrieves the failedJobs array from cache.

2. Fetches the first 5 sessions from the failedJobs array.

3. Dequeues sessions from the queue until the array length reaches 10.

4. Update each session's recording status via the EmpowerID API.

5. Converts the .guac videos to .mp4 using guacnec and ffmpeg utilities.

6. Retries failed jobs up to three times and update their status accordingly.

Uploader Service

The Uploader service is responsible for transferring the processed .mp4 files to a designated storage location, such as AWS or Azure.

2.1 CronJob: Session Checks & Uploads

This CronJob runs every 5 minutes to check for closed sessions and subsequently upload them.

The workflow for the CronJob is as follows:

1. Read file names from the /recording directory.

2. Use the EID API endpoint to fetch each session's storage mode and recording status.

3. Filter out sessions that are not marked as "Processed."

4. Add these sessions to the queue for uploading.

5. Update each session's recording status to "Uploading."

6. Transfer the files to the specified storage mode, handling any failures as needed.