FTP Job

FTP Job

An FTP Job is an automated task that transfers files between your Edge device and an external FTP, FTPS, or SFTP server. FTP Jobs make it easy to synchronize, collect, archive, or distribute files with remote systems - such as lab instruments, legacy file servers, or enterprise applications -using secure and robust file transfer workflows.

FTP Jobs operate on the built-in Edge storage of your device, powered by a Blob Storage Module (offering functionality similar to Azure Storage Accounts). This enables reliable file handling and transfer, even during intermittent network connectivity.

Key Features

  • Automated File Transfer: Send or retrieve files between Edge storage and remote FTP/FTPS/SFTP servers.

  • Edge-Based Operation: Runs locally on the Edge device for efficient and reliable file handling, including during temporary network outages.

  • Flexible Scheduling: Define how often the job runs using cron-style expressions or simple intervals.

  • Pattern Matching: Select which files to transfer based on patterns (e.g., by extension, by folder, or by naming conventions).

  • Advanced File Filtering: Filter files based on size, age, or modification date/time for precise control over what gets transferred.

  • Robust Operations: Automatically extract zip files and handle post-transfer cleanup (e.g., delete source files after transfer).

  • Secure Transfers: Full support for FTPS (FTP over TLS) and SFTP (SSH File Transfer Protocol) for encrypted communication.

  • Cloud Integration: Optionally send files to cloud storage (files-drop container) for centralized ingestion and processing or send files to a local container at the Edge for further processing.

Typical Use Cases

  • Automated Ingestion: Pull data files generated by lab equipment or other OT systems into the cloud for analysis.

  • Retrieve Measurements from files Write a custom module, that processes file from a container and send them to cloud ingestion. The FTP Job can find and prepare the files that needs to be processed.

  • Report Distribution: Periodically upload production reports or logs to central servers or cloud storage.

  • Data Archiving: Automatically backup important files from FTP servers to secure cloud locations.

  • File Synchronization: Keep remote folders and Edge storage in sync for distributed workflows or disaster recovery.

Configuring an FTP Job

Add a New Job

  1. Select the Area node where you want to add the job.

  2. Go to the Jobs tab.

  3. Click + New job, select FTP job, and provide a name for the job.

    Note: FTP Job is only available if there is a configured FTP Data Connector on the Area.

Create a FTP Job

Configure Job Settings

Configuration of FTP Job

  • Job Name: Provide a descriptive name (e.g., MoveLabFtpJob).

  • Enabled: Toggle to activate or deactivate the job.

  • Schedule (CRON): Define how frequently the job should run (e.g., every 10 minutes: 0/10 * * * *).

The CRON string consists of 5 characters separated by spaces, describing how often the schedule should be triggered on the Edge device.

Examples:

  • * * * * * Run every minute

  • */30 * * * * Run every 30 minutes

  • 0 * * * * Run every hour

  • 0 0 * * * Run once a day at midnight UTC

  • 0 0 1 * * Run on the 1st of each month at midnight UTC

  • 0 0 * * 1 Run at midnight every Monday UTC

See cron.help for more details.

FTP Settings

  • FTP Server: Choose the configured FTP/FTPS/SFTP data connector to use.

  • Operation: Select the operation mode, such as Upload to FTP Server, Download from FTP Server or Synchronize FTP server folder.

  • FTP Sync Folder: The source folder on the FTP server to monitor (e.g., import).

  • File Patterns: Define patterns to target files to handle using file system globbing patterns, you can define rules for files that should be targeted for the operation. (e.g., **/* for all files in any subdirectory).

    You can use multiple rules by seperating them with ;; Eg. (filepattern1);;(filepattern2);;(filepattern3) Here are some filepattern examples:

    • *.txt Matches all files with the file extension .txt in the folder.

    • *.* Matches all files regardless of file extension in the current folder.

    • *word* Matches all files that has the name 'word' in the filename.

    • styles/*.css Matches all files with extension '.css' in the directory 'styles/'.

    • scripts/*/* Matches all files in 'scripts/' or one level of subdirectory under 'scripts/'

    • images*/* Matches all files in a folder with name that is or begins with 'images'.

    • **/* Matches all files in any subdirectory.

    • dir/**/* Matches all files in any subdirectory under 'dir/'.

    • GTRS*/**/* Matches all files, that has a starting directory that starts with 'GTRS' and include all files from any subdirectory.

    For more information see Microsoft documentation which include more examples.

  • Extract ZIP Files: (Optional) Automatically extract files with .zip extension in transfer.

  • Post File Sync: Choose what happens to source files after transfer (e.g., Move synced files to another ftp folder , delete synced files from ftp folder, keep files (May result in re-download on next job trigger).

Advanced File Filtering

Optional Advanced File Filtering options

  • File Size Range: Restrict by minimum and/or maximum file size (e.g., only process files between 1 MB and 1 GB).

  • File Age: Restrict to files older/newer than a specific age (e.g., only process files older than 1 minute).

  • File Time Range: Restrict to files created/modified within a specified UTC time window.

Destination

  • Destination for Files: Choose the final destination, e.g., Send files to cloudor Send to blob storage container on device. The first option witll send files to a central ingestion storage account in the cloud. The latter option can be used if you want to further process the files before sending them to cloud.

  • Destination Folder: (Optional) Specify a target folder or use directory tokens for dynamic paths. Eg. folder1 or folder1/folder2 to create a deeper directory structure. You can also use tokens to build up the path using the tokens below:

    $(deviceid)

    Replace a part of a filepath with the Device Id of the Edge device using the token $(deviceid).

    rootfolder/$(deviceid)/otherfolder rootfolder/EdgeDevice-1/otherfolder

    $(yyyy)

    Replace a part of a filepath with the current Year UTC (Eg. 2021) using the token $(yyyy).

    rootfolder/$(yyyy)/otherfolder rootfolder/2021/otherfolder

    $(MM)

    Replace a part of a filepath with the current Month UTC (Eg. 01-12) using the token $(MM).

    rootfolder/$(MM)/otherfolder rootfolder/07/otherfolder

    $(dd)

    Replace a part of a filepath with the current Day of month UTC (Eg. 01-31) using the token $(dd).

    rootfolder/$(dd)/otherfolder rootfolder/07/otherfolder

    $(HH)

    Replace a part of a filepath with the current UTC hour (Eg. 00-24) using the token $(HH).

    rootfolder/$(HH)/otherfolder rootfolder/13/otherfolder

    $(mm)

    Replace a part of a filepath with the current UTC minute (Eg. 00-59) using the token $(mm).

    rootfolder/$(mm)/otherfolder rootfolder/57/otherfolder

    $(ss)

    Replace a part of a filepath with the current UTC Second (Eg. 00-59) using the token $(ss).

    rootfolder/$(ss)/otherfolder rootfolder/35/otherfolder

    $(FFF)

    Replace a part of a filepath with the current UTC MilliSecond (Eg. 000-999) using the token $(FFF).

    rootfolder/$(FFF)/otherfolder rootfolder/358/otherfolder

    $(unixts)

    Replace a part of a filepath with the current Unix Timestamp UTC (Number of milliseconds that has elapsed since 1970-01-01) using the token $(unixts).

    rootfolder/$(unixts)/otherfolder rootfolder/1629098246320/otherfolder

Example FTP Job Configuration

Setting
Example Value

Job Name

MoveLabFtpJob

Enabled

Yes

CRON Schedule

0/10 * * * *

FTP Server

FtpServer

Operation

Synchronize FTP server folder

FTP Sync Folder

import

File Patterns

**/*.sen;;**/*.int

Extract Zip Files

No

Post File Sync

Delete files from FTP folder when synced

Destination for Files

Send files to cloud

Destination Folder

$(deviceid)

Best Practices

  • Use descriptive names for jobs and folders to aid in troubleshooting and system management.

  • Schedule jobs to fit your workflow and avoid overloading the Edge device or the FTP server.

  • Test file patterns and destination settings before deploying to production.

  • Use secure protocols (FTPS or SFTP) and validate certificates for sensitive data transfers.

  • Enable post-transfer cleanup to avoid cluttering source folders and minimize reprocessing.

  • Monitor job execution via the File Transfer Module Jobs Tab for error tracking.

PreviousFile JobNextFile Share Job

Last updated

Was this helpful?