File Job

File Job

A File Job is an automated task that manages files locally on your Edge device, allowing you to copy, move, or extract files between storage containers or folders. File Jobs are ideal for organizing, processing, or preparing files before they are ingested, transferred, or archived - completely independent of external servers or network shares.

File Jobs operate on the built-in Edge storage of your device, which is powered by a Blob Storage Module (offering functionality similar to Azure Storage Accounts). This enables robust and flexible file handling, even during network outages.

Key Features

  • Edge-Local Operation: Runs entirely on the Edge device, ensuring fast, reliable file movement with no dependency on external connectivity.

  • Flexible Scheduling: Define how often the job runs using cron-style expressions or simple intervals.

  • Container/Folders: Move, copy, or extract files between any storage containers or folders configured on the Edge device.

  • Pattern Matching: Select files to handle based on patterns (e.g., all files in any subfolder use **/*, or target files of a specific extension in the import folder and any sub-folder where file extension is .csv use import/**/*.csv).

  • Zip Extraction: Automatically extract files from zip archives as part of your workflow.

  • File Operations: Choose to copy or move files, preserving or removing the originals as needed.

Typical Use Cases

  • Staging Files: Prepare and organize files collected from lab instruments or production systems before they are transferred to the cloud or another destination.

  • Automated Extraction: Unzip archived measurement files for further processing or ingestion.

  • Data Preparation: Filter and sort files into different containers or folders for downstream applications or analytics.

  • Edge Processing: Enable local processing of files for performance, compliance, or offline operations.

Configuring a File Job

  1. Add a New Job

    • Select the Area node where you want to add the job.

    • Go to the Jobs tab.

    • Click + New job and select File job and provide a name for the job.

    Create a File Job

  2. Configure Job Settings

    Configuration of File Job

    • Job Name: Provide a descriptive name (e.g., EdgeFileJob).

    • Enabled: Toggle to activate or deactivate the job.

    • Schedule (CRON): Define how frequently the job should run (e.g., every 10 minutes: 0/10 * * * *).

      The CRON string consists of 5 characters separated by spaces, describing how often the schedule should be triggered on the Edge device.

      Examples:

      • * * * * * Run every minute

      • */30 * * * * Run every 30 minutes

      • 0 * * * * Run every hour

      • 0 0 * * * Run once a day at midnight UTC

      • 0 0 1 * * Run on the 1st of each month at midnight UTC

      • 0 0 * * 1 Run at midnight every Monday UTC

      See cron.help for more details.

  3. Source

    • Source Container Name: The name of the storage container or folder where the files are located (e.g., tmplabfiles).

    • Source Folder: (Optional) Specify a subfolder if needed. Eg. folder1 or folder1/folder2 to create a deeper directory structure.

    • File Patterns: Define patterns to target files to handle using file system globbing patterns, you can define rules for files that should be targeted for the operation. (e.g., **/* for all files in any subdirectory).

      You can use multiple rules by seperating them with ;; Eg. (filepattern1);;(filepattern2);;(filepattern3) Here are some filepattern examples:

      • *.txt Matches all files with the file extension .txt in the folder.

      • *.* Matches all files regardless of file extension in the current folder.

      • *word* Matches all files that has the name 'word' in the filename.

      • styles/*.css Matches all files with extension '.css' in the directory 'styles/'.

      • scripts/*/* Matches all files in 'scripts/' or one level of subdirectory under 'scripts/'

      • images*/* Matches all files in a folder with name that is or begins with 'images'.

      • **/* Matches all files in any subdirectory.

      • dir/**/* Matches all files in any subdirectory under 'dir/'.

      • GTRS*/**/* Matches all files, that has a starting directory that starts with 'GTRS' and include all files from any subdirectory.

      For more information see Microsoft documentation which include more examples.

    • Extract Zip Files: Enable to automatically extract files with a .zip extension as part of the job.

    • File Operation: Choose between Copy (preserve originals) or Move (remove from source after processing).

  4. Destination

    • Destination Container Name: Target storage container for the moved/copied files (e.g., unzipped).

    • Destination Folder: (Optional) Subfolder for organization. Eg. folder1 or folder1/folder2 to create a deeper directory structure. You can also use tokens to build up the path using the tokens below:

      $(deviceid)

      Replace a part of a filepath with the Device Id of the Edge device using the token $(deviceid).

      rootfolder/$(deviceid)/otherfolder rootfolder/EdgeDevice-1/otherfolder

      $(yyyy)

      Replace a part of a filepath with the current Year UTC (Eg. 2021) using the token $(yyyy).

      rootfolder/$(yyyy)/otherfolder rootfolder/2021/otherfolder

      $(MM)

      Replace a part of a filepath with the current Month UTC (Eg. 01-12) using the token $(MM).

      rootfolder/$(MM)/otherfolder rootfolder/07/otherfolder

      $(dd)

      Replace a part of a filepath with the current Day of month UTC (Eg. 01-31) using the token $(dd).

      rootfolder/$(dd)/otherfolder rootfolder/07/otherfolder

      $(HH)

      Replace a part of a filepath with the current UTC hour (Eg. 00-24) using the token $(HH).

      rootfolder/$(HH)/otherfolder rootfolder/13/otherfolder

      $(mm)

      Replace a part of a filepath with the current UTC minute (Eg. 00-59) using the token $(mm).

      rootfolder/$(mm)/otherfolder rootfolder/57/otherfolder

      $(ss)

      Replace a part of a filepath with the current UTC Second (Eg. 00-59) using the token $(ss).

      rootfolder/$(ss)/otherfolder rootfolder/35/otherfolder

      $(FFF)

      Replace a part of a filepath with the current UTC MilliSecond (Eg. 000-999) using the token $(FFF).

      rootfolder/$(FFF)/otherfolder rootfolder/358/otherfolder

      $(unixts)

      Replace a part of a filepath with the current Unix Timestamp UTC (Number of milliseconds that has elapsed since 1970-01-01) using the token $(unixts).

      rootfolder/$(unixts)/otherfolder rootfolder/1629098246320/otherfolder

Example File Job Configuration

Setting
Value
Example

Job Name

EdgeFileJob

EdgeFileJob

Enabled

Yes

CRON Schedule

Every hour

0 * * * *

Source Container Name

tmplabfiles

tmplabfiles

Source Folder

(empty)

File Patterns

All files

**/*

Extract Zip Files

Yes/No

Yes

File Operation

Move

Move

Destination Container Name

unzipped

unzipped

Destination Folder

Name of Edge device

$(deviceid)

Best Practices

  • Use clear names for jobs, containers, and folders to simplify troubleshooting and management.

  • Schedule jobs appropriately to match your workflow and avoid unnecessary load on the Edge device.

  • Test with sample files to verify pattern matching and operations before deploying in production.

  • Remember: File Jobs cannot interact with external systems- use FTP Jobs or File Share Jobs for networked file transfers.

  • Monitor File Jobs Each time the File Job triggers, it will create a log entry in the File Transfer Module. You can monitor all Job Logs in the FileTransferModule Jobs Tab (the Asset Hierarchy containing the Job must be deployed first).

PreviousJobsNextFTP Job

Last updated

Was this helpful?